Add mart for OCW resources#2236
Conversation
There was a problem hiding this comment.
Pull request overview
This PR is intended to introduce a dimensional-layer-backed OCW resources mart (per PR description), and begins surfacing additional OCW resource metadata fields as scalar columns for downstream analysis.
Changes:
- Added scalar extraction of several resource-level metadata fields (license, description, file_type/size, ocw_type, external link status/wayback URL, audience, level) to
int__ocw__resources. - Updated intermediate and marts YAML model documentation (including adding a new
marts__ocw_resourcesmodel entry) and cleaned upcourse_leveldescription formatting.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/ol_dbt/models/marts/ocw/_marts__ocw__models.yml |
Adds schema/docs for marts__ocw_resources (but currently missing the corresponding model SQL in-repo). |
src/ol_dbt/models/intermediate/ocw/int__ocw__resources.sql |
Extracts additional scalar fields from websitecontent_metadata JSON into dedicated columns. |
src/ol_dbt/models/intermediate/ocw/_int_ocw__models.yml |
Documents the new extracted columns on int__ocw__resources and fixes course_level description wrapping. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| - name: marts__ocw_resources | ||
| description: OCW course resources (files, external resources, video, image) for | ||
| review and analysis | ||
| columns: |
| - dbt_utils.unique_combination_of_columns: | ||
| combination_of_columns: |
| @@ -0,0 +1,45 @@ | |||
| select | |||
There was a problem hiding this comment.
Not sure if you meant to leave out the config block here, it will use the project's default materialization (likely view). This should be fine since dim_ocw_resource is already a table, just wanted to note it.
There was a problem hiding this comment.
Yes, this was intentional. The config blocks on a few other marts are grants for sensitive certificate/profile data, which this mart does not need.
quazi-h
left a comment
There was a problem hiding this comment.
Builds ran clean for me, code looks good. I posted one comment but it is not a blocker.
What are the relevant tickets?
Part of https://github.com/mitodl/hq/issues/9943.
Description (What does it do?)
This PR adds a dimensional-layer-backed mart for OCW resources, replacing the Superset dataset that currently reads
int__ocw__resourcesdirectly. It addsdim_ocw_resource(one row percourse_uuid/resource_uuid, sourced fromint__ocw__resources) andmarts__ocw_resources(references only the dimensional layer, notint__*). It also surfaces nine fields from the raw resourcemetadataJSON as scalar columns:resource_license,resource_description,resource_file_type,resource_file_size,resource_ocw_type,resource_audience,resource_level,external_resource_status,external_resource_wayback_url, and drops the rawmetadatablob from the dim/mart. A few of these (audience,level,wayback_url) are currently sparse, but are expected to be filled in over time.How can this be tested?
First, run
filling in
<your name>as appropriate.Then, run
Finally, smoke-test this in Starburst Galaxy (https://mitol.galaxy.starburst.io/query-editor) by running a query such as